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Problem we are addressing 

■ We are in the second year of our initiative and studying 

> Parameters that affect the results of inspection 

> The relation between V&V effectiveness in early lifecycle (e.g., 
inspection) and late (testing) 

■ We are using this information to provide feedback and decision 
support to NASA projects, on questions such as: 

> Can I get guidance on how to plan 
my inspections based on results 
from projects like my own? 

> Based on my inspection results, 
what are the implications for the effort 
required to be spent on other non-optional 
activities, like system testing? 
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Our approach 


Literature Recommendations 


Historical Baseline Models 


Current Model Formulation 


I nputs from: 

❖gsfc cr) 

❖ GRC 
❖J PL 
❖j SC 

❖ MSFC 
More to come. . . 




Outputs: 

❖Automated feedback 
❖What if Analysis 
❖ Experience Bases 
❖T rends 



Users: 

❖ Projects 

❖ SEPG 

❖ inspection Planners 

❖ Researchers 
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First year results 

■ Collected more than 2,529 inspection records in our database 

> Evaluated old classification schema 

> Developed new classification based on existing standards and the 
collected data 

> Mapped data into new classification schema 


Developed prototype tool to support planning and reporting 

> Incorporated latest analyses and models based on the data 

> Designed capabilities for accepting data from various forms (e.g., 
JPL forms) as well as various databases 

> Gained feedback on usability and possible enhancements 


Created central inspection experience base 



> Provides materials necessary for applying inspections in various 
contexts: e.g., defect type definitions, mapping to various 
taxonomies, checklists, forms, ... 
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Unifying different defect classifications 

■ Motivation : Valuable defect data has been collected over the 
years across many Centers and projects 


issue : Different defect classifications used in historic and 
contemporary data sets, as well as across and within Centers 


Action : Define a unified defect classification schema along with 
a mapping to existing data sets 



Benefits : 

> Leverages data required by NPR 7150.2 for analysis and feedback 
to teams 

> Enables monitoring and validation of existing guidelines 

> Unified classification schema is applicable to inspections and testing 
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Mapping the different data sets 


historic data sets 


actions 


contemporary data 
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Analyze historical categories; 
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Define initial new categorization schema 
(i.e., mix of historic and common categorization schema) 


initial ODC-based new schema 
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Map historical data to new categorization, for categories 
that exist in both. 
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Mapping algorithm 
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Updating existing inspection guidelines 

Motivation : NASA guidelines for effective inspections 
(e.g., 3 points of control) were formulated in early 1990’s 

■ Issue. Development procedures (e.g., standards, languages, 
etc.) have changed over time; 

-> New factors must be considered 

■ Action : 

> Validate guidelines based on a wider set of recent data; 

> Refine the guidelines if needed (e.g., by adding more variables, 
tailoring to different domains, etc.) 

> Integrate them into an inspection support tool and training courses 

Benefits : Refined guidelines will increase effectiveness of 
inspections and provide better user guidance 


© 2008 Fraunhofer USA Inc. SAS_08_Full Life-cycle Defect Managementjnspections & tests_Shull 09/2008 Technical Presentation 8 






Fraunhofer USA, Inc 

Center for Experl mental 
Software Engineering 
Maryland 



User guidance based on heuristics 


optimum 
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Example: Comparison of team size 

Target team size: optimal is 4 to 6; borderline is 3 or 7 



requirements design source code test other 
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Comparing test and inspection data 

Motivation: Better knowledge of inspection’s strengths & weaknesses 
could be used to better allocate resources among V&V activities. 

Issue : Defects that slip through inspections aren’t found until much later; 
different defect type descriptors mean they often are hard to compare. 

Action. Compare test and inspection defect profiles (on the same 
projects or within the same domain) 

Benefits: Past knowledge about recurring defect types can be used to 
select the right overall strategy for optimal V&V planning 


Research Questions: 


What defects types are typically removed by inspections vs. testing? 

What project characteristics (size, language, software domain, new 
development/enhancements) influence the types of defects found? 



What percent of logic errors can be expected to be removed by 

inspections? 

Can test results be used for post-mortem analysis of 
inspection performance? 
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Overview: Inspections vs. testing 
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Initial results: Across projects 
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Research Question: What defect types are typically removed by 

inspections vs. testing? In this domain: 

-> Overall the defect removal profile seems similar, but 

-> I nspections found on average 64% of the total system defects 
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Initial results: Within a project 
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Research Question: What defect types are typically removed by 
inspections vs. testing? Specifically, for a maintenance project: 

-> Many more internal interface defects were found by inspections 


© 2008 Fraunhofer USA Inc. SAS_08_Full Life-cycle Defect Managementjnspections & tests_Shull 09/2008 Technical Presentation 14 


Fraunhofer USA, Inc 


Center for Experl mental 
Software Engineering 
Maryland 


Improving tool support 

■ Motivation : Data and resources from across NASA, that use 
different taxonomies, cannot easily be leveraged without 
centralized tool support. 

■ Issue : Need to do mappings and analysis without requiring extra 
steps from the user, and to seamlessly integrate the results. 

■ Action : 

> Centralize existing materials and resources -> Experience Base; 

> Integrate Experience Base and results data into a combined 
dashboard 

■ Benefits: Integrating real-time feedback into normal engineering 
activities, for: 

> The planning of inspections, 

> Collection of data, 

> Analysis and building of up-to-date baselines, 

> Feedback and improvement. 
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Providing an inspection experience base 



Choose subc ate gory: | Requirements 
Choose type: [Aii 
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Table of Contents 


About this Experience Base 
General Terms & Acronyms 
SI Tools & Resources 
AH 


Filter by title: 

Filter by description: 
Filter by date: 

7 item(s) found 



Checklists 

Defect classifications 
Forms & Templates 
Tools 

Inspections Repository 
£ Training & Services 
Inspection Tutorial 
Full EB Access 
£ Suggest new documents 


Contact EB manager: 
fs hnO@fc-md. mud, edn 
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Improving tool support for inspections 
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Future work 


Refine the test and inspection data comparison 

> Obtain additional data sets for testing and refining our preliminary conclusions 

> Integrate test results into inspection tool 

Initial deployment of tool 

> Obtain additional feedback on usability and future deployment 

> Pursue expansion of the Experience Base with testing-related materials 
-> a centralized site for V&V resources 

■ Integrating with other existing inspection data forms and tool 
support 

> Especially eRoom-based tool available through Kevin Carmichael / GRC 
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Contact information 


Forrest Shull (PI) Sally Godfrey (NASA POC) 

fshull@fc-md.umd.edu sara.godfrey@nasa.gov 
301 - 403-8970 301 - 286-5706 

Myrna Regardie 

mregardie@fc-md.umd.edu 

301 - 403-2050 



Inspection Experience Base on-line at: 
http://fc-md.umd.edu/EB 
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